{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise - Hyperparameter Tuning with Grid Search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise you will train a base model and then try to find better combinations of hyperparameter values using the grid search technique."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - imports\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Execute the cells below to read prepared data on the [Invesco QQQ Trust, Series 1 (NASDAQ: QQQ)](https://finance.yahoo.com/quote/QQQ/) ETF from 1999 to 2017. We have already engineered some technical indicators as features and cleaned the data. The DataFrame also includes the raw level of the VIX (Volatility Index)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - load data and display basic statistics\n",
    "df = pd.read_csv(\"Data.csv\")\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'd like to try and predict the direction of 5-day future returns. Run the cell below to split the data and prepare for model training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - Define features and target and split data\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X = df.drop(columns=[\"fut_ret_5d_is_pos\", \"Date\"])\n",
    "y = df[\"fut_ret_5d_is_pos\"]\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Training a base model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train a `RandomForestClassifier` and train it using its default hyperparameter values. As this is a tree-based model, you do not need to scale the features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - imports\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "# FILL IN - Instantiate a RandomForestClassifier and fit it to the training data\n",
    "# Use random_state=52 for reproducibility\n",
    "# Set n_jobs=-1 to enable parallel processing using all available CPU cores\n",
    "clf = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will focus on precision as our performance metric, as we would like to avoid False Positives as much as possible.  \n",
    "Below, we have provided a function that plots 5-fold cross-validated precision scores. Study it and invoke it to plot the learning curves using the training set. You should be able to observe that the model is overfitting to the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - imports\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import learning_curve\n",
    "\n",
    "# DO NOT MODIFY - plotter function\n",
    "def plot_learning_curve(model, X, y, cv=5, n_jobs=-1):\n",
    "    train_sizes, train_scores, test_scores = learning_curve(\n",
    "        model,\n",
    "        X,\n",
    "        y,\n",
    "        cv=cv,\n",
    "        n_jobs=n_jobs,\n",
    "        scoring=\"precision\",\n",
    "    )\n",
    "    train_scores_mean = np.mean(train_scores, axis=1)\n",
    "    test_scores_mean = np.mean(test_scores, axis=1)\n",
    "    plt.plot(train_sizes, train_scores_mean, label=\"CV training score\")\n",
    "    plt.plot(train_sizes, test_scores_mean, label=\"CV test score\")\n",
    "    plt.title(\"Learning curve for Random Forest Classifier\")\n",
    "    plt.xlabel(\"Training examples\")\n",
    "    plt.ylabel(\"Precision\")\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "# FILL IN - Plot the learning curve for the RandomForestClassifier using the training data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What was the average cross-validated precision score on the training set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - imports\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# FILL IN - Get the 5-fold cross-validated precision of the classifer on the training data\n",
    "# Use n_jobs=-1 for parallel processing\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And how does this score compare to the precision on the test set? - **HINT:** Use the fitted classifier's `predict()` method to get an array of predictions on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - imports\n",
    "from sklearn.metrics import precision_score\n",
    "\n",
    "# FILL IN - Get the precision of the classifier on the test data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Grid search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that you can use the `get_params()` method of the classifierto see a list of its hyperparameters and other settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we have picked 3 different values for each of the 4 major hyperparameters of `RandomForestClassifier`. Using Scikit-Learn's `GridSearchCV` class, perform a 5-fold cross-validated grid search using the provided search grid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT MODIFY - imports\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "# DO NOT MODIFY - the `hyperspace` of hyperparameters to search\n",
    "search_grid = {\n",
    "    'n_estimators': [50, 100, 200],\n",
    "    'max_depth': [5, 10, 20],\n",
    "    'min_samples_split': [2, 7, 15],\n",
    "    'min_samples_leaf': [1, 2, 4]\n",
    "}\n",
    "\n",
    "# FILL IN - Instantiate a GridSearchCV object with the fitted RandomForestClassifier, the search grid, 5-fold cross-validation, and precision scoring.\n",
    "# Fit it to the training data\n",
    "# Don't forget to set n_jobs=-1 for parallel processing. This may take a minute or two even with parallel processing.\n",
    "search = ...\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Store the best parameters, best score, and best estimator (model). (These are attributes of `search`.) Feel free to print out the best CV precision score. How does it compare to the base model? Which combination of values yielded this result?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN - Get the best parameters, best score, and best estimator from the GridSearchCV object\n",
    "best_params = ...\n",
    "best_score = ...\n",
    "best_estimator = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below to see the top 5 results in detail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search_results = pd.DataFrame(search.cv_results_)\n",
    "search_results.sort_values('rank_test_score').head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Re-use the same `plot_learning_curve()` function we provided earlier to plot the learning curve for the best estimator found using training data. How does it compare to the learning curve of the original classifier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN - Plot the learning curve for the best estimator using the training data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, use this estimator to evaluate the test set and get the new test performance score. How does it compare?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FILL IN - Get the precision of the best model on test data\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
